Image processing apparatus, image taking apparatus and program

ABSTRACT

An image processing apparatus, which obtains a synthesized image whose exposure is corrected by synthesizing a first image and a second image, is disclosed. The image processing apparatus comprises a detection section which detects an amount of displacement of the second image with respect to the first image which is a reference image, a coordinate conversion section which performs coordinate conversion to the second image so as to conform to the first image, and a synthesis section which synthesizes the second image having been subjected to the coordinate conversion with the first image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 10/852,334,filed May 24, 2004 the entire disclosure of which is hereby incorporatedby reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage taking apparatus such as a digital camera which improves imagetaking accuracy by correcting image blur caused by vibration and aprogram which is used for the apparatuses.

2. Description of Related Art

Operations important for image taking such as exposure determination andfocusing, etc., of cameras that are currently used, have been completelyautomated, and even a person who is unskilled in camera operations isunlikely to cause an image taking failure.

Furthermore, recently, a system that represses image blur from beingcaused by vibration applied to the camera has also been studied, andfactors that cause a photographer to fail in image taking have beenreduced to almost zero.

Herein, a vibration isolation system that represses image blur isbriefly described.

Camera shake when image taking appears as vibration with a frequency of,normally, 1 Hz through 10 Hz, and for enabling the camera to take apicture without image blur even when such vibration occurs at a point ofexposure, it is required that camera shake due to vibration is detectedand a correcting lens is displaced within an optical axis orthogonalplane according to the results of this detection (optical vibrationisolation system).

Namely, in order to take a picture without image blur even when camerashake occurs, it becomes necessary that, first, camera shake isaccurately detected, and second, an optical axis change due to vibrationis corrected.

Camera shake can be detected by, in principle, mounting on a camera avibration detecting unit that detects acceleration, angularacceleration, angular velocity, and angular displacement by means of alaser gyro, etc., and carries out appropriate calculation for theresults of this detection. Then, by driving a correcting optical unitthat makes an image taking optical axis eccentric on the basis of thedetection information on camera shake outputted from the vibrationdetecting unit, image blur correction is carried out.

On the other hand, Japanese Patent Publication No. 3110797 discloses amethod in which image taking is repeated a plurality of times in anexposure period with a length that does not cause vibration, and aplurality of images obtained through the image taking are synthesizedwhile correcting image divergence among the images to obtain a takenimage (synthesized image) of a long exposure period.

Recent digital cameras have become smaller in size than silver haloidcompact cameras, and in particular, a camera that has an image pickupdevice of a VGA class has been downsized so that it is built-in aportable electronics device (such as a portable phone). In order tomount the abovementioned optical vibration isolation system on a camera,it is necessary that the vibration correcting optical unit is madesmaller or the vibration detecting unit is made smaller.

However, in the vibration correcting optical unit, since a correctinglens must be supported and highly accurately driven, there is a limit todownsizing. In addition, most of the vibration detecting units that arecurrently used utilize inertia, so that if the vibration detecting unitsare downsized, detection sensitivity lowers and accurate vibrationcorrection becomes impossible.

Furthermore, vibration to be applied to cameras includes angularvibration around a predetermined axis and shifting vibration that shakesa camera parallel, and although the angular vibration is correctable bythe optical vibration isolation system, the shifting vibration is hardlycorrected. This shifting vibration tends to become greater as the camerabecomes smaller.

On the other hand, as a different vibration isolation system, asemployed in a video camera for taking a moving image, a method in whicha motion vector of an image plane is detected based on an output of animage pickup device and an image readout position is changed accordingto the detected motion vector to obtain a moving image without imageblur can also be employed. This method has an advantage in that thecamera can be downsized since the vibration detecting unit and thecorrecting lens in the abovementioned optical vibration isolation systembecome unnecessary.

However, this vibration isolation system used in video cameras cannot beeasily applied to digital cameras. The reason for this is describedbelow.

Motion vector extraction in a video camera is carried out for each imagereading, and for example, when images of 15 frames are extracted persecond, a motion vector is detected by comparing these extracted images.

However, in a case where a still image is taken by a digital camera,exposure is carried out only once for an object to be taken, so thatmotion vector detection through comparison of images as in a videocamera is not possible. Therefore, the vibration isolation system forvideo cameras cannot be simply applied to digital cameras.

On the other hand, in the vibration isolation method disclosed inJapanese Patent Publication No. 3110797, since image taking is repeateda plurality of times, an image taking period becomes long. The longimage taking period does not pose a problem when an image of a stillobject is taken. However, when an image of an object such as a personthat even slightly moves is taken, shake of an object side (objectvibration) is caused, and image blur caused by object vibration cannotbe suppressed although image blur caused by hand vibration can besuppressed, and a taken image may deteriorate.

SUMMARY OF THE INVENTION

According to one aspect of an image processing apparatus of the presentinvention, the image processing apparatus which obtains a synthesizedimage whose exposure is corrected by synthesizing a first image and asecond image which are taken successively by using an image pickupdevice, comprises a detection section which detects an amount ofdisplacement of the second image with respect to the first image whichis a reference image, and a coordinate conversion section which performscoordinate conversion to the second image so as to conform to the firstimage on the basis of the detection result of the detection section, andfurther comprises a synthesis section which synthesizes the second imagehaving been subjected to the coordinate conversion with the first image,and a controller which sets an image taken by using light from anelectric flash to the first image.

According to one aspect of an image taking apparatus of the presentinvention comprises the image processing apparatus and an image pickupdevice.

According to one aspect of a program of the present invention, theprogram which obtains a synthesized image whose exposure is corrected bysynthesizing a first image and a second image which are takensequentially by using an image pickup device, comprises a detecting stepof detecting an amount of displacement of the second image with respectto the first image which is a reference image; a coordinate conversionstep of performing coordinate conversion to the second image so as toconform to the first image on the basis of the detection result in thedetecting step; and a synthesizing step of synthesizing the second imagehaving been subjected to the coordinate conversion with the first image;and a setting step of setting an image taken by using light from anelectric flash to the first image.

The characteristics of the image processing apparatus, the image takingapparatus and the program of the present invention will be clarified bythe following detailed description of embodiments with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a camera of Embodiment 1 of the invention.

FIG. 2 is a drawing for explaining coordinate conversion for two images.

FIGS. 3A and 3B are drawings for explaining characteristic pointextraction regions.

FIG. 4 is an explanatory view of image synthesis.

FIG. 5 is a flowchart showing image taking operations in the camera ofEmbodiment 1.

FIG. 6 is an explanatory view of a characteristic point extractionregion of Embodiment 2 of the invention.

FIG. 7 is a timing chart showing image taking processing operations inthe camera of Embodiment 2.

FIG. 8 is a flowchart showing image taking operations in the camera ofEmbodiment 2.

FIG. 9 is a block diagram of a camera of Embodiment 3 of the invention.

FIG. 10 is a flowchart showing image taking operations in the camera ofEmbodiment 3.

FIG. 11 is a block diagram of a camera of Embodiment 4 of the invention.

FIG. 12 is a flowchart showing image taking operations in the camera ofEmbodiment 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1

FIG. 1 is a block diagram showing a structure of a camera (image takingapparatus) of Embodiment 1 of the invention. A light flux (image takinglight) that has entered from an image taking lens 11 passes through ashutter 12 and is restricted in light amount by a stop 13, and thenreaches an image pickup device 15. The image pickup device 15 comprisesa semiconductor image pickup device such as an MOS or a CCD. The imagepickup device 15 is driven by an image pickup device drive circuit 16which has received a control signal from an image taking control circuit111.

The image taking lens 11 moves on an optical axis L by receiving adriving force from an AF drive motor 14, and then stops at apredetermined focus position. Thereby, the focusing operation is carriedout. The AF drive motor 14 drives in response to a drive signal from afocus drive circuit 19.

The stop 13 has a plurality of stop blades, and these stop bladesactuate by receiving a driving force from a stop drive unit 17 andchange an aperture area (stop aperture diameter) of a light passingaperture. The shutter has a plurality of shutter blades, and theseshutter blades open and close the light passing aperture by receiving adriving force from a shutter drive unit 18. Thereby, the light amount ofthe light flux to enter the image pickup device 15 can be controlled.

Driving of the focus drive circuit 19, the stop drive unit 17, theshutter drive unit 18, and the illumination unit 10 is controlled by animage taking control circuit 111. Herein, the illumination unit 10irradiates an object with illumination light by receiving a controlsignal from the image taking control circuit 111.

The image taking control circuit 111 detects an object luminance(photometry) on the basis of an output of a signal processing circuit112 that is described later, and on the basis of the results of thephotometry, the stop aperture diameter of the stop 13, the open periodof the shutter 12, and use of the illumination unit 10 are determined.The image taking control circuit 111 calculates a focus position on thebasis of the output of the signal processing circuit 112 while drivingthe focus drive circuit (image taking lens 11). Namely, the camera ofthis embodiment carries out focusing of the image taking optical systemaccording to a so-called contrast detection method.

Analog signals outputted from the image pickup device 15 are convertedinto digital signals by an A/D converter 110 and inputted into thesignal processing circuit 112. The signal processing circuit 112 appliessignal processing such as luminance signal and color signal formation tothe inputted signals to generate color video signals.

Then, image signals subjected to the signal processing in the signalprocessing circuit 112 are displayed as a taken image by being outputtedto a display unit 116, and outputted to and recorded in a recordingcircuit 117.

The operations described above are carried out when an image of anobject having a brightness that does not require vibration correction istaken. On the other hand, to take an image of a dark object, an exposureperiod lengthens, and image blur may be caused by vibration. In thiscase, a photographer operates an operation switch (vibration isolationswitch) that is provided on the camera, not shown, whereby the vibrationisolation system is turned on and switches into the following operation.

First, when a photographer depresses halfway a release button (notshown) provided on the camera, an image taking preparation operation(including focusing and photometry, etc.) is started. The image takingcontrol circuit 111 determines the open period (exposure period) of theshutter 12 and the stop aperture diameter of the stop 13 on the basis ofphotometric values obtained through photometry. Herein, generally, anobject is dark in a case where a vibration isolation system is used, thestop is set to full-open and the exposure period is set to long periodof exposure.

In the camera of this embodiment, this long period of exposure isdivided into a plurality of short exposure periods, and the image pickupoperation is continuously repeated the number of times corresponding tothe number of divisions. Due to this division into short exposureperiods, each image obtained through the exposure becomes underexposed,however, these images are less influenced by vibration (have less imageblur). Then, the plurality of image data are synthesized into one afterfinishing all image pickup operations, whereby data of oneexposure-corrected image can be generated.

However, when data of a plurality of images are generated through aplurality of times of image pickup operations, even if there is noinfluence of vibration on each image, the image composition (objectimage) may be slightly displaced among the images due to vibrationduring the plurality of times of image pickup operations. Herein, if thedata of the plurality of images are synthesized as they are, thesynthesized image may blur according to the composition displacementamong the images.

In this embodiment, a plurality of image signals outputted from theimage pickup device 15 in response to a plurality of serial image pickupoperations are subjected to signal processing by the signal processingcircuit 112 after being converted into digital signals by the A/Dconverter circuit 110 as mentioned above. Output signals from the signalprocessing circuit 112 are inputted into an image storage circuit 113 aswell as the image taking control circuit 111.

The image storage circuit 113 stores all the data of the plurality ofimages obtained through the series of image pickup operations.

A displacement detection circuit 114 (detection section) acquires imagedata stored in the image storage circuit 113 and extracts acharacteristic point (specific point) from an image region, andcalculates the position coordinates of the extracted characteristicpoint within the image taking plane.

Herein, a case where a picture of a person standing against a buildingis taken through a plurality of serial image pickup operations isconsidered. In some cases where vibration is applied to the cameraduring a plurality of serial image pickup operations, as shown in theframe 119 b of FIG. 2, an image whose composition is displaced from theimage of the frame 119 a is generated.

The displacement detection circuit 114 extracts an edge 123 a of awindow 122 a having a high luminance from the building 121 a positionedby the side of the person 120 a in the image region of the frame 119 a.The displacement detection circuit 114 also extracts a characteristicpoint 123 b from the frame 119 b in the same manner as in the case ofthe abovementioned frame 119 a. Then, the displacement detection circuit114 compares the characteristic point 123 a with the characteristicpoint 123 b and corrects the difference between these. Namely, thedisplacement detection circuit 114 applies coordinate conversion to theimage data of the frame 119 a or the frame 119 b so that thecharacteristic point 123 a and the characteristic point 123 b overlapeach other.

For example, the edges 123 a and 123 b are made correspondent to eachother by means of template matching, etc., and the characteristic point123 b that corresponds to the characteristic point 123 a in the frame119 b is searched for, whereby the displacement amount between thecharacteristic point 123 a and the characteristic point 123 b iscalculated, and coordinate conversion is carried out on the basis of thecalculated displacement amount.

In this embodiment, as shown in FIG. 2, coordinate conversion is appliedto the image data of the frame 119 b so that the characteristic point123 b of the frame 119 b is moved in the direction of the arrow 124 ofFIG. 2 so as to overlap the characteristic point 123 a of the frame 119a.

Herein, an explanation is given for the reason why the peripheral regionat the periphery of the central region in the region of the imageacquired through image pickup operations is used as a region from whicha characteristic point is extracted. In many cases of image taking, amain object is positioned around the center of an image taking plane andthe main object is a person. In such a case, if the region correspondingto the main object is selected as a characteristic extraction region, aproblem occurs due to movement of the main object.

During a plurality of serial image pickup operations, image blur iscaused not only by vibration applied to the camera but also by movementof the main object. Herein, in the case where a characteristic point isextracted from the region corresponding to the main object, imagecoordinate conversion is carried out on the basis of the movement of themain object. In this case, it seems that an image without vibration onthe main object can be created, however, movement of a person as a mainobject is generally complicated, so that vibration of the main objectmay not be properly corrected depending on the position where acharacteristic point is selected, as described below.

For example, in a case where a characteristic point is extracted from aregion corresponding to the eye of a main object (person), blinkinginfluences and obstructs proper correction for the actual vibration ofthe entire object. Also, if a characteristic point is extracted from aregion corresponding to the tip of a hand, the hand easily moves, sothat coordinate conversion is carried out differently from the propercoordinate conversion for the actual vibration of the entire object.

As mentioned above, even when a characteristic point is extracted from aregion corresponding to a person and image coordinate conversion iscarried out on the basis of the extracted characteristic point,coordinate conversion is not always properly carried out for the person.Even after the images whose coordinates have not been properly correctedare synthesized, the points of the image other than the characteristicpoint are left displaced, so that an image with less image blur cannotbe obtained.

Therefore, as in this embodiment, by extracting a characteristic pointfrom a region corresponding to a still object such as a background,proper coordinate conversion can be applied to the entire image, so thata preferable synthesized image with less image blur can be obtained.However, in the characteristic point extraction method of thisembodiment, object vibration influences come out as mentioned above.

In this embodiment, the object is irradiated with illumination light bydriving the illumination unit 10 only at a predetermined number-thoperation of the plurality of serial image pickup operations. Namely, asdescribed below, influence of object vibration is suppressed by usingthe illumination light of the illumination unit 10.

In the description given below, an image obtained by using theillumination unit 10 is defined as a first image, and another image (orother images) obtained without use of the illumination unit 10 isdefined as a second image. Herein, the following difference existsbetween the first and second images in addition to the abovementionedcomposition displacement. Namely, the brightness of an object region(illuminated region) which the illumination light has reached in thefirst image is different from the brightness of a region thatcorresponds to the abovementioned illuminated region in the secondimage.

In the first image, in the main object region which the illuminationlight has reached, sufficient exposure is obtained, however, exposure isinsufficient in the region (background) which the illumination lightdoes not reach. The reason for this is that the main object such as aperson is generally positioned near the camera and illumination lightreaches the main object, and the background is positioned far from thecamera and the illumination light does not reach the background.

Herein, for the underexposed region (background), coordinate conversionis carried out to conform the second image to the first image and thefirst image and the second image whose coordinates have been convertedare synthesized, whereby underexposure is compensated.

FIG. 3A and FIG. 3B are drawings for explaining a selection method for acharacteristic point extraction region by the displacement detectioncircuit 114. FIG. 3A shows a first image obtained by using theillumination unit 10, and FIG. 3B shows a second image obtained withoutuse of the illumination unit 10.

In the first image 125 (FIG. 3A), illumination light reaches a person120 a, so that almost proper exposure is obtained for the person 120 a.On the other hand, at a background other than the person 120 a,underexposure occurs since the illumination light does not reach.

The second image 126 (FIG. 3B) is underexposed for the person 120 b andthe background since no illumination light is provided.

Comparing the first image 125 and the second image, only the brightnessat the person is different, and other regions have no difference inbrightness.

The background region which have no difference in brightness (that is,the difference of the brightness is smaller than a predetermined value)becomes underexposed since illumination light does not reach. Therefore,in this embodiment, the underexposed region is regarded as a point ofimage synthesis (exposure correction) and set as a characteristicextraction region.

Namely, in the camera of this embodiment, on the basis of theabovementioned peripheral regions in the images obtained through theimage pickup operations and a region whose brightness hardly differsbetween the first and second images, a characteristic point extractionregion is determined.

In the images shown in FIG. 3A and FIG. 3B, the edges 123 a and 123 b ofthe window having a comparatively high luminance are extracted ascharacteristic points in the region which have no difference inbrightness, that is, the region corresponding to the building. Then, asdescribed in FIG. 2, the characteristic point 123 a of the first image125 and the characteristic point 123 b of the second image 126 arecompared with each other, and a difference between these is corrected(coordinate conversion).

The coordinate conversion circuit 115 (coordinate conversion section)shown in FIG. 1 applies coordinate conversion to the second image 126 sothat the characteristic point 123 b of the second image 126 overlaps(conforms) the characteristic point 123 a of the first image 125. Thedisplacement detection circuit 114 extracts the characteristic pointfrom the remaining second images in the same manner as that of thesecond image 126. The coordinate conversion circuit 115 appliescoordinate conversion to the remaining second images so that thecharacteristic points of the remaining second images overlap thecharacteristic point 123 a of the first image 125.

In the abovementioned description, characteristic point changes arefound by calculating the coordinates of the characteristic point of eachimage, however, in actuality, correlation calculation of the first image125 and the second image 126 is carried out, and pixel changescorresponding to each other are regarded as a motion vector and definedas a characteristic point change. Furthermore, for the remaining secondimages, by means of correlation calculation with respect to the firstimage 125, characteristic point changes are also determined.

In the description given above, the case where one characteristic pointis extracted, however, it is also possible that a plurality ofcharacteristic points are extracted. In this case, an average motionvector or the minimum scalar of the plurality of characteristic pointscan be regarded as a characteristic point change.

Herein, use of the abovementioned minimum value of a characteristicpoint change is for selecting the most moveless characteristic pointsince characteristic points extracted in the characteristic pointextraction region have the possibility of moving by themselves.

The coordinate conversion circuit 115 applies coordinate conversion toeach image data (second images) according to the characteristic pointchanges determined by the displacement detection circuit 114. Data ofthe respective images subjected to coordinate conversion by thecoordinate conversion circuit 115 are outputted to an image synthesiscircuit 118 (synthesis section) and synthesized into one image data.

In the camera of this embodiment, the first image obtained by using theillumination unit 10 is defined as a reference (center) image for imagesynthesis, and the second image data are subjected to coordinateconversion so that the second images overlap the first image.

Herein, the reason why the first image 125 is set as a reference imageis described.

As shown in FIG. 2, in a case where two image data whose compositionsare displaced from each other are synthesized, as shown in FIG. 4, aregion 127 in which the two images do not overlap each other isgenerated. Therefore, the image synthesis circuit 118 cuts the region127 and applies expansion and complement processing to the region inwhich the two images overlap each other, whereby the synthesized imageis restored to the original image size. In this expansion and complementprocessing, a part of the image is cut according to the orientation anddegree of the composition displacement.

Herein, among the first and second images, the most accurate imageinformation (image information on the main object) is obtained from thefirst image that have been obtained by using the illumination unit 10.

Therefore, in order to avoid cutting a part of the first image, it ispreferable that the first image is defined as a reference image and thesecond images are made to overlap the first image.

In a case of a digital image, exposure correction is possible by raisingthe gain of even only one underexposed image data, however, if the gainis raised, noise also increases and results in an undesirable image.However, in a case where the gain of the entire image is raised bysynthesizing a plurality of images as in this embodiment, noise of theimages is averaged, so that an image with a high S/N ratio can beobtained, and as a result, exposure can be made proper while suppressingnoise. In another consideration, it can also be said that, for example,data of a plurality of images are taken by allowing noise and setting ahigh sensitivity of the image pickup device 15, and data of these imagesare subjected to averaging process, whereby random noise included in thesynthesized image is reduced.

The image data synthesized by the image synthesis circuit 118 isoutputted to the display unit 116, and displayed as a still image andrecorded in the recording circuit 117.

FIG. 5 is a flowchart showing image taking operations in the camera ofthis embodiment, and this flow starts when the vibration isolationswitch is operated (turned on).

In Step S1001, the process waits until the sw1 is turned on byhalf-depression of the release button by a photographer, and when thesw1 is turned on, the process advances to Step S1002.

In Step S1002, light from an object is made to reach a light receivingsurface of the image pickup device 15. Thereby, accumulated electriccharge is readout from the image pickup device 15 according to thereceived light amount. The image taking control circuit 111 drives theAF drive motor 14 to move the image taking lens 11 in the direction ofthe optical axis L while detecting the contrast of the object image onthe basis of the output from the signal processing circuit 112.

Then, when the contrast becomes highest, driving of the image takinglens 11 is stopped, whereby the image taking optical system is in-focuscondition (that is, focusing by a hill-climbing method).

Focusing can also be carried out by a phase-difference detecting method.The image taking control circuit 111 measures the brightness of theobject on the basis of the output of the image pickup device 15(photometry).

In Step S1003, based on the brightness of the object obtained in StepS1002, the number of times of image pickup operations (reading imagesignals from the image pickup device 15) is calculated. Herein, forproper exposure based on the photometry results, the stop 13 is set tofull-open (for example, f2.8), and the open period (exposure period) ofthe shutter 12 is set to ⅛ seconds.

Herein, when the focal length of the image taking optical system is 30mm as regards 35 mm film, there is a possibility that image blur iscaused by vibration in the exposure period of ⅛ seconds, so that theexposure period is set to 1/32 seconds and the image pickup operation iscarried out four times. Namely, by increasing the number of times ofimage pickup operations according to the shortened exposure period, thetotal exposure period is made to be almost equal.

On the other hand, when the focal length of the image taking opticalsystem is 300 mm, the exposure period is set to 1/320 seconds and theimage pickup operation is carried out forty times so as to repress imageblur.

In Step S1004, on the display unit provided within the finder of thecamera or a liquid crystal display unit provided on the outer package ofthe camera, information on the number of times of image pickupdetermined in Step S1003 is displayed. A photographer can confirm thenumber of times of image pickup operation by looking at the indicationon the display unit. It is also possible that a photographer is informedof the number of times of image pickup operation by using voice, etc.

In Step S1005, the process waits until the sw2 is turned on byfull-depression of the release button. When the half-depression of therelease button is released during waiting in Step S1005, that is, whenthe sw1 is turned off, the process returns to start.

In Step S1006, the image taking control circuit 111 judges whether ornot the image pickup operation is the first time. Herein, in the casewhere the first image pickup operation is carried out, the processadvances to Step S1008. When the image pickup operation is not the firstoperation, the process advances to Step S1007.

In Step S1007, by carrying out the image pickup operation withoutdriving the illumination unit 10, second image data are generated, andthe process advances to Step S1009.

In Step S1008, by carrying out the image pickup operation by driving theillumination unit 10, first image data is generated, and the processadvances to Step S1009.

In Step S1009, image data obtained in Step S1007 or Step S1008 (firstimage data or second image data) is stored in the image storage circuit113.

In Step S1010, it is judged whether or not the image pickup operationhas been carried out the number of times determined in Step S1003, andthe process waits while circulating Step S1006, Step S1007 and StepS1009 until all the number of times of image pickup operations arecompleted. Then, when all the number of times of image pickup operationsare completed, the process advances to Step S1011.

In Step S1011, the displacement detection circuit 114 extracts acharacteristic image (characteristic point) from the first image and thesecond images by the abovementioned method, and the coordinates of theextracted characteristic point within the image pickup region arecalculated.

In Step S1012, the second image data are subjected to coordinateconversion by the coordinate conversion circuit 115. In detail, thesecond image data are subjected to coordinate conversion so that thecharacteristic points of the second images overlap the characteristicpoint of the first image as mentioned above. Herein, in Step S1011, in acase where the process advances to Step S1012 after the characteristicpoint extraction from the first image and coordinate calculation of thecharacteristic point, coordinate conversion is not applied to the firstimage data. The reason for this is that the first image data is areference image for coordinate conversion.

In Step S1013, it is judged whether or not all the second image datahave been subjected to coordinate conversion. If all second image datahave not been subjected to coordinate conversion, the process circulatesSteps S1011 and S1012 until application of coordinate conversion to allsecond image data is completed. On the other hand, when application ofcoordinate conversion to all second image data is completed, the processadvances to Step S1014.

Namely, in the processes from Step S1011 through Step S1013 mentionedabove, first, the processing of Step S1011 is applied to the first imagedata, and then the processing of Step S1011 and Step S1012 is applied toeach of the plurality of second image data. Herein, the second imagedata that have been coordinate-converted are stored in the image storagecircuit 113.

In Step S1014, the first image data and the plurality of second imagedata are synthesized by the image synthesis circuit 118. Herein, imagesynthesis is carried out by the averaging process of coordinate signalscorresponding to the image data, and random noise included in the imagesis reduced through the averaging process. Then, the gain of the imagewith reduced noise is raised to make exposure proper.

In Step S1015, a region in the synthesized image data obtained in StepS1014 (corresponding to the region 127 of FIG. 4) in which the images donot overlap each other due to image blur is cut. Then, the synthesizedimage data is subjected to expansion and complement processing so thatthe cut synthesized image is restored to the original image size.

In Step S1016, the synthesized image data obtained in Step S1015 isoutputted to a liquid crystal display unit mounted on the back surface,etc., of the camera and displayed as a still image. Thereby, aphotographer can observe the synthesized image.

In Step S1017, the synthesized image data obtained in Step S1015 isrecorded on a recording medium which is composed of, for example, asemiconductor memory and is attachable to and detachable from thecamera. Thereby, the image taking operation is completed.

In Step S1018, the process returns to start.

In a case where the release button has been depressed halfway and thesw1 has been turned on at Step S1018, the process advances in the flowin sequence again, to Steps S1001, S1002, S1003, and S1004. In a casewhere the release button has been fully depressed and the sw2 has beenturned on at Step S1018, the process does not return to start but waitsat Step S1018.

In the description mentioned above, a characteristic point is extractedfrom image information, and displacements in the images (displacementsof the second images from the first image) are corrected on the basis ofthe characteristic point changes, however, the image displacementcorrection method is not limited to the abovementioned method.

For example, the following method can be used. First, the first andsecond images are divided into a plurality of regions without condition,and differences (motion vectors) between the divided regions in thefirst image and the divided regions in the second image corresponding tothe abovementioned divided regions in the first image are calculated.Then, among the plurality of motion vectors, based on a motion vectorwith a high frequency, coordinate conversion is applied to the secondimage, and the second image is conformed to the first image.

The abovementioned motion vector with a high frequency is not based onthe object's vibration, but based on image blur caused by vibration(camera shake), so that the image blur caused by vibration can also beaccurately corrected by the abovementioned method.

Embodiment 2

A camera of Embodiment 2 of the present invention is a modified exampleof the camera of the above-mentioned Embodiment 1. Herein, the structureof the camera of this embodiment is almost the same as that described inEmbodiment 1 (FIG. 1).

In Embodiment 1, a characteristic point extraction region is set to theabovementioned peripheral region in the entire region of an imageobtained through an image pickup operation.

However, the characteristic point extraction region is not limited tothe abovementioned peripheral region, and a region other than a focusarea set within the image taking plane may be regarded as acharacteristic point extraction region, or a region other than a regionincluding a focus area which is currently focused among a plurality offocus areas may be regarded as a characteristic point extraction region.

Normally, image taking operation is carried out in a condition where afocus area overlaps a main object (person), so that in order to extracta characteristic point from a region other than a region correspondingto the main object, a region other than the focus area is set as acharacteristic point extraction region.

FIG. 6 shows a characteristic point extraction region within an imagetaking plane.

FIG. 6 shows a condition where, among focus areas 128 a, 128 b, 128 c,128 d, and 128 e set within an image taking plane (frame 120 a), a mainobject is focused in the focus area 128 c. Herein, the main objectregion 130 is a region having a specific extent around the focus area128 c which is focused, and is a region in which an object is highlylikely positioned. Furthermore, the peripheral region 131 shaded in FIG.6 excludes the main object region 130 in the image taking region, and isa region in which a still object is highly likely positioned. In thisembodiment, the peripheral region 131 is set as a characteristic pointextraction region.

In this embodiment, depending on which of the focus areas 128 a, 128 b,128 c, 128 d, and 128 e the main object is focused in, the main objectregion 130 and the peripheral region 131 (characteristic pointextraction region) are changed.

An appropriate image is extracted as a characteristic point from theperipheral region 131, coordinate conversion is applied to the secondimage data on the basis of the coordinates of the extractedcharacteristic point, and the first image data and thecoordinate-converted second image data are synthesized, whereby an imagewith less image blur can be obtained.

Instead of image data synthesis after completion of coordinateconversion for all image data (second image data) as shown in FIG. 5, itis also possible that after coordinate conversion is applied to onesecond image data, image synthesis is applied to thecoordinate-converted second image data.

FIG. 7 shows a timing chart when the image synthesis is carried outevery time coordinate conversion is applied to the second image data asmentioned above.

In response to exposure f1, a signal subjected to photoelectricconversion and charge accumulation at the image pickup device 15 isreadout as an image pickup signal F1 (first image data). Then, alongwith reading-out of an image pickup signal F2 (second image data),correlation calculation of the image pickup signal F1 and the imagepickup signal F2 is carried out. Thereby, a characteristic point changebetween the two images is determined, and the two image pickup signalsF1 and F2 are synthesized to obtain a synthesized signal C2.

Next, along with reading-out of an image pickup signal F3 (second imagedata), correlation calculation of the synthesized signal C2 and theimage pickup signal F3 is carried out. Thereby, a characteristic pointchange between the two images (the synthesized image and the readoutimage) is determined, and the synthesized signal C2 and the image pickupsignal F3 are synthesized to obtain a synthesized signal C3.

Next, along with reading-out of an image pickup signal F4, correlationcalculation of the synthesized signal C3 and the image pickup signal F4is carried out. Thereby, a characteristic point change between the twoimages (the synthesized image and the readout image) is determined, andthe synthesized signal C3 and the image pickup signal F4 are synthesizedto obtain a synthesized signal C4.

Then, the obtained synthesized signal C4 (synthesized image data) isoutputted to the liquid crystal display unit provided on the backsurface, etc., of the camera and displayed as a taken image, andrecorded on a recording medium.

FIG. 8 is a flowchart of the operations described in FIG. 7. Incomparison with the flowchart of FIG. 5, the image storage processing ofStep S1009 is eliminated.

In FIG. 8, after coordinate conversion (Step S2010), image synthesis(Step S2011) is carried out, and it is judged whether or not the imagepickup operation has completed the number of times of image pickupoperation determined in Step S2003 (Step S2012). Then, in a case whenthe image pickup operation does not complete all the number of times,the next image pickup operation is carried out (Step S2006), and whenall the number of times of image pickup operations are completed,expansion and complement processing is carried out (Step S2013).

Herein, the operations of Step S2001 through Step S2008 are the same asthe operations of Step S1001 through Step S1008 of FIG. 5. Theoperations of Step S2013 through Step S2016 are the same as theoperations of Step 1015 through Step S1018 of FIG. 5.

In this embodiment, as described in FIG. 7, each reading-out of imagedata from the image pickup device, the readout image data is synthesizedwith image data readout in advance or synthesized image data issynthesized in advance. Therefore, only one image data always exists,and it becomes unnecessary to store a plurality of image data.

Namely, since the synthesized image data is renewed every time imagedata is readout, it becomes unnecessary to store a plurality of imagedata as in Embodiment 1. Therefore, the camera of this embodiment doesnot have the image storage circuit 113 shown in FIG. 1.

In the flowchart shown in FIG. 8, it appears that the next image pickupoperation is not carried out until processing of all images is completedin Step S2012, however, in actuality, as shown in the timing chart ofFIG. 7, output of an image pickup signal (reading-out of image data),correlation calculation, and image synthesis are simultaneously carriedout.

As described above, in Embodiment 1 and Embodiment 2 of the presentinvention, reading-out of image data in a short exposure period in whichvibration hardly influences is repeated a plurality of times, andcoordinate conversion and synthesis are applied to an obtained pluralityof image data, whereby one taken image data (synthesized image data) isgenerated. Namely, by synthesizing a plurality of image data,underexposure can be complemented. In addition, by applying coordinateconversion to the plurality of image data (specifically, second imagedata), composition displacement in each image data caused by vibrationcan be corrected, whereby a taken image with less image blur can beobtained.

Thereby, although the camera is a digital still camera, it canelectronically correct image blur as in the case of a video camera.Therefore, exclusive members (correcting lens, etc.) for image blurcorrection as provided in a silver haloid camera become unnecessary, sothat the camera is downsized. Furthermore, in Embodiments 1 and 2, sincedisplacement of the image itself is corrected, not only angularvibration but also shifting vibration can be corrected.

Furthermore, in the abovementioned Embodiments 1 and 2, in a case wherecoordinate conversion is applied to image data on the basis ofcharacteristic point displacement in each image, it is considered whichof regions in the image pickup plane the characteristic point isextracted from.

As shown in FIG. 6, in a case where the image pickup plane is dividedinto a main object region 130 in which a main object (person) is highlylikely positioned and the remaining peripheral region 131, if acharacteristic point is extracted from the main object region 130 andcomposition displacement is corrected, as mentioned above, thecomposition displacement cannot be accurately corrected due to vibrationof the person himself/herself.

Therefore, in Embodiment 2, a characteristic point is extracted from theperipheral region 131, and based on the extracted characteristic point,composition displacement of each image is corrected(coordinate-converted). Then, the images whose coordinates have beenconverted are synthesized. Thereby, the composition displacement basedon a still object other than a person can be corrected, and one imagewithout image blur (taken image whose exposure has been corrected) canbe obtained.

Furthermore, in the camera of Embodiment 2, by using illumination lightof the illumination unit 10, a region whose brightness is differentbetween the first image and the second image and a region whosebrightness is almost the same between the first image and the secondimage (that is, a region in which the difference of the brightness issmaller than a predetermined value) are created, and the region whosebrightness is almost the same is set as a characteristic pointextraction region. Namely, in the camera of this embodiment, on thebasis of the abovementioned peripheral region 131 and the region with abrightness difference, a characteristic extraction region is determined.

Thereby, a characteristic point can be accurately extracted from aregion corresponding to a still object. In addition, by synthesizing theimages upon applying coordinate conversion to the second image data onthe basis of the extracted characteristic point, an image (synthesizedimage) with less image blur can be obtained.

Embodiment 3

FIG. 9 is a block diagram of Embodiment 3 of the present invention. Apoint of difference of the camera of this embodiment from the camera ofEmbodiment 1 is that a brightness adjusting circuit (adjusting section)21 is provided. For the same members as described in Embodiment 1, thesame reference numerals are used.

Herein, the role of the brightness adjusting circuit 21 is described.The brightness adjusting circuit 21 darkens, in the second image whosecoordinates have been converted by the coordinate conversion circuit115, an image in a region (brightness adjusting region serving as afirst region) which corresponds to a region (illuminated region) of thefirst image is sufficiently irradiated with illumination light of theillumination unit 10. In detail, the gain of a luminance signal in thebrightness adjusting region is lowered, or the brightness adjustingregion is blackened (condition without image information).

In the region (illuminated region) which is sufficiently irradiated withillumination light of the illumination unit 10 in the first image,proper exposure has already been obtained by only the first image. Then,if the first image data and the second image data are synthesized in allthe regions, exposure becomes excessive in the illuminated region. Or, abackground with a high luminance may overlap a person irradiated withillumination light.

Therefore, by darkening the regions of the second images that correspondto the illuminated region, only the first image data is used in theilluminated region in this embodiment.

Herein, the reason why a background with a high luminance is avoidedfrom overlapping a person is described in detail.

In the first image, since the person is irradiated with illuminationlight of the illumination unit 10, proper exposure is obtained for theperson. In the second images, the person is not irradiated withillumination light of the illumination unit 10, so that the objectbecomes dark. In some cases of the dark object, the person conceals abackground with a high luminance.

In such a case, due to composition displacement caused by vibration, thebackground with a high luminance may be displaced from the person andseen from the edge of the person. Then, if the background (image) with ahigh luminance appearing from the edge of the person is superposed onthe first image, the background with a high luminance that is concealedby the person and cannot be seen in the first image overlaps the personin the first image and appears (for example, from the outline of theperson), and this greatly deteriorates the accuracy of the image of theperson as the main object.

In this embodiment, in order to avoid the abovementioned problem, thebrightness in the regions of the second image corresponding to theilluminated region are lowered by the brightness adjusting circuit 21.

By comparing the first image with the second image, a region whosebrightness is different between the first image and the second image (aregion sufficiently irradiated with illumination light in the firstimage) is set As a region to be adjusted in brightness. The brightnessadjusting circuit 21 adjusts the brightness in a region (brightnessadjusting region) of the second image corresponding to the illuminatedregion of the first image.

Herein, the brightness adjusting region (region corresponding to theilluminated region of the first image) is determined on the basis of thesecond image data whose coordinates have been converted by thecoordinate conversion circuit 115.

In some cases, the second images before being subjected to coordinateconversion have compositions displaced from the first image due tovibration, etc. Namely, when the object part on a specific coordinate inthe first image and the object part at the same coordinates on thespecific coordinate in the second image before being subjected tocoordinate conversion are compared with each other, the parts are notthe same.

Therefore, in this embodiment, the brightness of the brightnessadjusting regions are adjusted in the second images whose compositiondisplacements have been corrected by coordinate conversion.

As a method of brightness adjustment for the second images, first, thebrightness adjusting regions in the second images are blackened(condition without image information). Then, the darkness of the imageat the boundary between the brightness adjusting region and a region(second region) other than the brightness adjusting region is changed instages or continuously. In detail, the luminance signal of the imagedata is gradually changed from the brightness adjusting region sidetoward the remaining region side.

The reason for this is that the boundary of the two regions (thebrightness adjusting region and the remaining region) becomesunnaturally conspicuous and preferable image can not be obtained if thebrightness adjustment is applied distinctly to the regions.

In the abovementioned Embodiments 1 and 2, the illumination unit 10 ismade to emit light at the first time of image pickup operation of theplurality of times of image pickup operations to be successively carriedout, however, in this embodiment, the illumination unit 10 is made toemit light for the last image pickup operation of the plurality of timesof image pickup operations.

Hereinafter, the reason why the illumination unit 10 is made to emitlight for the last image pickup operation is described.

When a person is set as a main object and the image taking operation isperformed, normally, the person thinks that image taking operation isfinished when emission of the illumination unit 10 is finished, andmoves from the image taking position immediately after emission isfinished.

As in the camera of this embodiment, when a plurality of image data areacquired through a plurality of times of image pickup operations, thetotal image pickup period lengthens, and if the illumination unit 10 ismade to emit light at the initial stage, a person as an object maygreatly move after emission.

Therefore, in this embodiment, the illumination unit 10 is made to emitlight when the last image pickup operation is carried out and no imagepickup operation is carried out after the emission. Thereby, until theplurality of serial image pickup operations are completed, movement ofthe object (person) can be suppressed.

Furthermore, in the image pickup operation for acquiring the first imagedata, by making the illumination unit 10 to emit light at the finalstage of exposure as in the case of rear-curtain flash synch, movementof the person can be effectively suppressed. Furthermore, it is alsopossible that emission (pre-emission) for acquiring information onregular emission with a proper emission light amount of the illuminationunit 10 can be carried out before regular emission.

The abovementioned pre-emission can be carried out immediately beforethe regular emission. In detail, pre-emission is carried out before theillumination unit 10 is made to emit light (regular emission) to acquirethe first image data. Furthermore, in the case of rear-curtain flashsynch, the pre-emission can be carried out at the final stage ofexposure immediately before regular emission. Furthermore, pre-emissioncan be carried out before the first image pickup operation of theplurality of times of serial image pickup operations, that is, beforeacquiring the first one of the plurality of second image data.

FIG. 10 is a flowchart showing image taking operations in the camera ofthis embodiment, and this flow starts when the vibration isolationswitch provided on the camera is turned on.

In Step S3001, the process waits until the sw1 is turned on byhalf-depression of the release button by a photographer, and when thesw1 is turned on, the process advances to Step S3002.

In Step S1002, exposure to the image pickup device 15 is started. Theimage taking control circuit 111 moves the image taking lens 11 in thedirection of the optical axis L by driving the AF drive motor 14 whiledetecting the contrast of an object image based on an output of thesignal processing circuit 112. Then, when the contrast of the objectimage reaches its peak, the image taking optical system is in-focusstate by stopping the movement of the image taking lens 11. The imagetaking control circuit 111 calculates the brightness of the object onthe basis of the output of the image pickup device 15 (photometry).

In Step S3003, the number of times of image pickup is determined on thebasis of the brightness of the object obtained in Step S3002. Herein,for proper exposure on the basis of the results of the photometry inStep S3002, the stop 13 must be set to full-open (for example, f2.8) andthe open period (exposure period) of the shutter 12 must be set to ⅛seconds.

Herein, when the focal length of the image taking optical system is 30mm as regards 35 mm film, image taking with an exposure period set to ⅛seconds may result in image blur due to vibration. Therefore, by settingthe exposure period to be shorter than ⅛ seconds, image blur isrepressed from appearing on an image obtained through exposure, and bycarrying out exposure a plurality of times, the total exposure period ismade almost equal to the abovementioned ⅛ seconds. In detail, theexposure period is set to 1/32 seconds and the image pickup operation isrepeated four times.

On the other hand, when the focal length of the image taking opticalsystem is 300 mm, the exposure period is set to 1/320 seconds and theimage pickup operation is repeated forty times so as to repress imageblur.

In Step S3004, information on the number of times of image pickupoperation obtained in Step S3003 is displayed on a display unit providedwithin the finder of the camera or a liquid crystal display unitprovided on the outer package of the camera. Thereby, a photographerconfirms the number of times of image pickup operation. The photographermay be informed of the number of times of image pickup operation byusing voice, etc.

In Step S3005, the process waits until the sw2 is turned on byfull-depression of the release button. In Step S3005, when thehalf-depression of the release button is released and the sw1 is turnedoff, the process returns to start.

In Step S3006, the image taking control circuit 111 judges whether ornot the current image pickup operation is the last image pickupoperation. Herein, in the case of the last image pickup operation, theprocess advances to Step S3008. When it is not the last image pickupoperation, the process advances to Step S3007.

In Step S3007, by carrying out the image pickup operation withoutemission of the illumination unit 10, the second image data are acquiredand the process advances to Step S3009.

In Step S3008, the first image data is acquired by carrying out theimage pickup operation by making the illumination unit 10 to emit light,and then the process advances to Step S3009.

In Step S3009, image data acquired through the image pickup operation ofStep S3007 or Step S3008 is stored in the image storage circuit 113.

In Step S3010, it is judged whether or not the image pickup operationhas been carried out the number of times determined in Step S3003.Herein, when all the image pickup operations have not been completed,the process waits while circulating Step S3006 and Step S3007 (or StepS3008). When all the image pickup operations are completed, the processadvances to Step S3011.

In Step S3011, in the displacement detection circuit 114, acharacteristic image (characteristic point) is extracted from aperipheral region of the first or second image, and the coordinates ofthe extracted characteristic point are calculated. In detail, asmentioned above, the first image and the second images are compared, anda characteristic point is extracted from a region (illuminated region)except for a region whose brightness differs for each image. Then, thecoordinates of each extracted characteristic point within the imagetaking plane are calculated.

In Step S3012, coordinate conversion is applied to each image data inthe coordinate conversion circuit 115. Herein, coordinate conversion isnot applied to the first image data and the first image data is set as areference image for coordinate conversion.

In actuality, correlation calculation of the first image data and onesecond image data is carried out to determine a characteristic pointchange. For the remaining second image data, correlation with the firstimage data stored in advance is also calculated to determinecharacteristic point changes.

In Step S3013, first, the position (coordinates) of a region(illuminated region) on the first image whose brightness differs whencomparing the first image and the second images, is calculated. Then,regions (brightness adjusting regions) at the same position in thesecond images subjected to coordinate conversion in Step S3012 as theilluminated region are darkened. In detail, as mentioned above, thebrightness adjusting regions in the second images are blackened(condition without image information), and the boundary between thebrightness adjusting regions and the remaining regions are shaded off(darkness of the images are changed in stages or continuously).

Herein, in a region of a synthesized image corresponding to theilluminated region, only the image information of the first image isused as mentioned above. Thereby, the synthesized image can be repressedfrom being deteriorated by using the image information of the secondimages in the illuminated region.

In Step 3014, it is judged whether or not the processing of Steps S3011through S3013 has been completed for all the second image data obtainedin Step S3007. Herein, until the processing of Steps S3011 through S3013is completed for all the second image data, the process returns to StepS3011 and repeats the processing of Steps S3011 through S3013. When theabovementioned processing is completed for all the second image data,the process advances to Step S3015.

In Step S3015, image synthesis process is applied to the first andsecond image data in the image synthesis circuit 118. Herein, in imagesynthesis process, coordinate signals corresponding to the respectiveimages (first and second image data) are subjected to averaging process,and random noise included in the images is reduced by averaging process.Then, the gain of image data with reduced noise is raised to makeexposure proper.

In Step S3016, in the synthesized image synthesized in the imagesynthesis circuit 118, a region (the region 127 of FIG. 4) in which theimages (first and second images) do not overlap due to compositiondisplacement is cut. Then, the synthesized image data is subjected toexpansion and complement processing so that the cut synthesized image isrestored to the original image size.

In Step S3017, the synthesized image data (taken image data) obtained inStep S3016 is outputted to a liquid crystal display unit provided on theback surface, etc., of the camera, and is displayed as a taken image onthe liquid crystal display unit.

In Step S3018, the synthesized image data obtained in Step S3016 isrecorded on a recording medium.

In Step S3019, the process returns to start. When the release button isdepressed halfway and the sw1 is on in Step S3019, the process advancesin the flow sequentially again, to Steps S3001, S3002, S3003, and S3004.When the release button is fully depressed and the sw2 is on in StepS3019, the process does not return to start but waits in Step S3019.

In the camera of this embodiment, image information of the first imageobtained by using illumination light of the illumination unit 10 is usedfor a main object (person), and for a background, images (first andsecond images) obtained through a plurality of times of image pickupoperations are synthesized to correct exposure. Thereby, an imageincluding a main object and a background both of which have beenproperly exposed can be obtained.

As described above, the camera of this embodiment has the followingeffect in addition to the effects described in Embodiments 1 and 2. Thatis, the brightness in the brightness adjusting regions in the secondimage data that have been subjected to coordinate conversion arelowered, only image information of the first image is used in theilluminated region, and the first and second image data are synthesizedin the illuminated region, whereby creation of an unnatural synthesizedimage can be repressed. Furthermore, by shading-off the boundaries ofthe illuminated regions and the remaining regions, the boundaries can berepressed from becoming unnatural.

Furthermore, after image pickup operations for acquiring the secondimage data are finished, an image pickup operation for acquiring thefirst image data is carried out, whereby all the image pickup operationscan be completed in time with a light emitting operation of theillumination unit 10, and this represses that the object (person) movesbefore completion of all image pickup operations.

Embodiment 4

The structure of a camera of Embodiment 4 of the present invention isshown in FIG. 11. Points of difference from Embodiment 3 (FIG. 9) arethat a brightness adjustment prohibiting circuit 31 (adjustmentprohibiting section) and an image synthesis prohibiting circuit 32(synthesis prohibiting section) are provided, and image data is inputtedfrom the image storage circuit 113 into the recording circuit 117.

Herein, an operation of the brightness adjustment prohibiting circuit 31is described. The brightness adjustment prohibiting circuit 31 operateswhen the first image data is acquired and an object is not sufficientlyirradiated with illumination light of the illumination unit 10. Forexample, in some cases where a person as an object is far from thecamera, illumination light of the illumination unit 10 does not reachthe person.

As in the abovementioned embodiment, the displacement detection circuit114 compares the first image and the second image, sets a region whichhas brightness difference between these as an illuminated region, andsets a region other than the illuminated region as a characteristicpoint extraction region.

However, in a case where the object is not sufficiently irradiated withillumination light of the illumination unit 10, the brightnessdifference between the first image and the second image is small. Thatis, the brightness difference between the first and second images issmaller than a predetermined value. Furthermore, when comparing thefirst image with the second image before being subjected to coordinateconversion, the brightness at the same coordinates may be slightlydifferent. In this embodiment, when the region with a brightnessdifference is small, that is, when the ratio of the region with abrightness difference to the entire image region is smaller than apredetermined value, it is judged that irradiation of the illuminationlight is not sufficient.

In the abovementioned case, the brightness in a region of the secondimage corresponding to the region slightly irradiated with theillumination light of the illumination unit 10 being darkened, exposurein this region becomes insufficient even after carrying out imagesynthesis.

The brightness adjustment prohibiting circuit 31 judges thatillumination by the illumination unit 10 is insufficient in many regionson the basis of the abovementioned results of the displacement detectioncircuit 114, and prohibits the brightness adjusting circuit 21 fromadjusting the brightness of the second image. In this case, imagesynthesis is applied to all regions of the first image and the secondimage. By thus synthesizing the first and second image data in allregions, exposure of a synthesized image can be made proper.

Next, an operation of the image synthesis prohibiting circuit 32 isdescribed.

As mentioned above, when an object is not sufficiently irradiated withillumination light of the illumination unit 10, the brightnessadjustment prohibiting circuit 31 prohibits the operation of thebrightness adjusting circuit 21.

On the other hand, in some cases, the illumination light of theillumination unit 10 does not illuminate a part of the image takingregion, but illuminates the entire image taking region. In this case,when the first and second image data are synthesized, the synthesizedimage becomes overexposed.

Therefore, the image synthesis prohibiting circuit 32 prohibits imagesynthesis in the image synthesis circuit 118 when the illumination lightof the illumination unit 10 illuminates the entire image taking region.

The displacement detection circuit 114 compares the first image and thesecond image, and judges a region which has brightness difference as aregion (illuminated region) irradiated with illumination light of theillumination unit 10. Then, a region other than the illumination regionis used as a characteristic point extraction region.

However, in a case where all objects within the image taking plane areclose to the camera, for example, the image taking operation isperformed with respect to a person close to the camera in his/herentirety or a case where the reflectance of a background is high, forexample, a white wall exists immediately behind a person, the differenceof the brightness occurs with respect to the entire image taking region.Namely, when comparing the first and second images, a region having nobrightness change between the images does not exist, and the differenceof the brightness occurs in all regions between the images.

In the displacement detection circuit 114, when it is judged that thedifference of the brightness occurs with respect to the entire imagetaking region, the image synthesis prohibiting circuit 32 judges thatthe image obtained by synthesizing the first image data and the secondimage data will result in overexposure by receiving the abovementionedinformation from the displacement detection circuit 114.

Herein, in the case where there is the difference of the brightness withrespect to the entire image taking region, the second images areblackened (condition without image information) by the processing of thebrightness adjusting circuit 31, and exposure difference does notoccurred even by image synthesis. However, small noise included in thesecond image data may deteriorate the synthesized image data due toimage synthesis, so that the image synthesis processing in the imagesynthesis circuit 118 is prohibited in this embodiment.

Next, a reason why image data is inputted from the image storage circuit113 into the recording circuit 117 is described. In Embodiment 1, onlythe image data obtained through image synthesis processing is recordedin the recording circuit 117, and the first and the second image datawhich have been used for the image synthesis are not recorded in therecording circuit 117 although they are temporarily stored in the imagestorage circuit 113.

In this embodiment, at least the first image data is recorded in therecording circuit 117 together with the synthesized image data.According to an object and its brightness, etc., a photographer may wantto use an image (first image) obtained by making the illumination unit10 to emit light or want to use a synthesized image obtained bysynthesizing a plurality of image data. Therefore, by recording thefirst image data and the synthesized image data in the recording circuit117, any of the images can be selected.

Herein, it is also possible that the second image data is recorded inthe recording circuit 117, however, in this case, the recording capacityof the recording circuit 117 is filled up with the second image data,and this reduces the possible number of times of image taking.

Therefore, in the camera of this embodiment, in addition to thesynthesized image data, the first image data is recorded in therecording circuit 117, and only one of the plurality of second imagedata can be recorded in the recording circuit 117 as appropriate.

FIG. 12 is a flowchart describing image taking operations of the cameraof this embodiment. This flow starts when the vibration isolation switchprovided on the camera is turned on.

In Step S4001, the process waits until the sw1 is turned on byhalf-depression of the release button by a photographer, and when thesw1 is turned on, the process advances to Step S4002.

In Step S4002, exposure to the image pickup device 15 is started. Theimage taking control circuit 111 moves the image taking lens 11 in thedirection of the optical axis L by driving the AF drive motor 14 whiledetecting the contrast of an object image based on an output from thesignal processing circuit 112. Then, when the contrast of the objectimage reaches its peak, the movement of the image taking lens 11 isstopped, whereby the image taking optical system is in-focus state.Then, the image taking control circuit 111 calculates the brightness ofthe object based on the output of the image pickup device 15(photometry).

In Step S4003, on the basis of the brightness of the object obtained inStep S4002, the number of times of image pickup operation is determined.Herein, for proper exposure based on the photometry results obtained inStep S4002, the stop 13 must be set to full-open (for example, f2.8),and the open period (exposure period) of the shutter 12 must be set to ⅛seconds.

Herein, when the focal length of the image taking optical system is 30mm as regards 35 mm film, image taking with an exposure period set to ⅛seconds may result in image blur due to vibration. Therefore, for imagetaking while suppressing image blur, the exposure period is set to 1/32seconds and the image pickup operation is set to be carried out fourtimes.

On the other hand, when the focal length of the image taking opticalsystem is 300 mm, for image taking while suppressing image blur, theexposure period is set to 1/320 seconds and the image pickup operationis set to be carried out forty times.

In Step S4004, information on the number of times of image pickupoperation determined in Step S4003 is displayed on a display unitprovided within the finder of the camera or a liquid crystal displayunit provided on the outer package of the camera. Thereby, aphotographer is informed of the number of times of image pickupoperation. It is also possible that voice, etc., is used to inform thephotographer of the number of times of image pickup operation.

In Step S4005, the process waits until the sw2 is turned on byfull-depression of the release button. When the half-depression of therelease button is released and the sw1 is turned off in Step S4005, theprocess returns to start.

In Step S4006, the image taking control circuit 111 judges whether ornot the process is at the last of the plurality of times of image pickupoperation. Herein, when the process is of the last image pickupoperation, the process advances to Step S4008. When the process is notof the last image pickup operation, the process advances to Step S4007.

In Step S4007, the second image data are acquired by carrying out imagepickup operation without emission from the illumination unit 10, and theprocess advances to Step S4009.

In Step S4008, the first image data is acquired by carrying out theimage pickup operation by making the illumination unit 10 to emit light,and the process advances to Step S4009.

In Step S4009, the image data acquired in Step S4007 and Step S4008 arestored in the image storage circuit 113.

In Step S4010, it is judged whether or not the image pickup operationhas been carried out the number of times of image pickup operationdetermined in Step S4003, and when all the image pickup operations havenot been completed, the process waits while circulating Step S4006 andStep S4007 (or Step S4008). When all the image pickup operations havebeen completed, the process advances to Step S4011.

In Step S4011, the displacement detection circuit 114 extracts acharacteristic image (characteristic point) from the first and secondimages and calculates the coordinates of the extracted characteristicpoint. In detail, as mentioned above, the first image and the secondimage are compared, a characteristic point is extracted from a regionother than a region with a brightness difference, and the coordinates ofthe extracted characteristic point in the image taking region arecalculated.

Furthermore, the displacement detection circuit 114 compares thebrightness in the first image and the brightness in the second image,and when a region whose brightness is much brighter in the first imagethan in the second image occupies a predetermined region or larger inthe entire region of the first image, for example, when the centralregion of the image is sufficiently bright, and almost 80% of theregions other than the central region is bright by illumination light,the brightness in the entire image region is judged as sufficient. Onthe other hand, with respect to the region with a brightness differencebetween the first image and the second image, when the brightness in thefirst image is insufficient, illumination is judged as insufficient.

In Step S4012, the displacement detection circuit 114 judges whether ornot the entire image region has a sufficient brightness as mentionedabove. When the entire image region has a sufficient brightness, theprocess advances to Step S4013, and when the brightness is insufficient,the process advances to Step S4019.

In Step S4013, coordinate conversion is applied to the second image databy the coordinate conversion circuit 115. Herein, the first image datais not subjected to coordinate conversion and is defined as a referenceimage for coordinate conversion.

In Step S4014, it is judged whether or not the brightness in the firstimage is insufficient, that is, whether or not the illumination light ofthe illumination unit 10 is insufficient. Herein, when the brightness inthe first image is insufficient, the process advances to Step S4016, andwhen the brightness is not insufficient, the process advances to StepS4015.

Namely, when the brightness in the region irradiated with illuminationlight in the first image is insufficient, processing to darken thebrightness in the second image is prohibited, and the first image dataand the second image data are synthesized, whereby proper exposure isobtained. In this embodiment, the brightness adjustment processing ofthe brightness adjusting circuit 21 is prohibited, however, according tothe condition of irradiation of the illumination light of theillumination unit 10, that is, the condition of the brightness in thefirst image, the level of the brightness adjustment processing may bechanged.

For example, when the brightness in the first image is sufficient, aregion of the second image corresponding to the region having asufficient brightness in the first image is blackened (condition withoutimage information). Furthermore, when the object is irradiated to someextent by illumination light of the illumination unit 10 although thebrightness in the first image is insufficient, a region of the secondimage corresponding to the region irradiated with the illumination lightin the first image is not blackened, but can be darkened to some degree.Furthermore, when the irradiation of the illumination light to theobject is equal to or lower than a predetermined amount, that is, whenthe object is not generally irradiated with the illumination light andthe brightness in the first image is insufficient, it is possible thatbrightness adjustment processing is not applied to the second imagedata, that is, the second images are not blackened.

In Step S4015, the brightness adjusting circuit 21 compares the firstimage data and the second image data and determines a region with abrightness that differs between these (illuminated region that isirradiated with illumination light). Then, a region (brightnessadjusting region) of the second image subjected to coordinate conversionin Step S4013, corresponding to the illuminated region, is darkened.

Herein, as a brightness adjusting method for the second image data, thebrightness adjusting regions in the second images are blackened(condition without image information), and the boundary between thebrightness adjusting regions and the remaining regions are shaded off.

Namely, data with respect to the brightness adjusting regions in thesecond image data are not used for image synthesis, and only data withrespect to the illuminated region in the first image data are used.Thereby, deterioration of the synthesized image which occurs by usingdata of the brightness adjusting regions of the second image data forimage synthesis can be suppressed.

In Step S4016, it is judged whether or not the processing of Steps S4011through S4015 has been completed for all the first and second imagedata. Herein, until the processing is completed for all the image data,the process returns to S4011 and repeats the processing. When theprocessing is completed for all the image data, the process advances toStep S4017.

In Step S4017, the first and second image data are synthesized by theimage synthesis circuit 118. Herein, the image synthesis is carried outby averaging of the coordinate signals of the respective image data, andrandom noise in the images is reduced by averaging process. Then, thegain of the image with reduced noise is raised to make exposure proper.

In Step S4018, a region (region 147 of FIG. 4) in which the images donot overlap due to composition displacement in the image synthesized bythe image synthesis circuit 118 is cut. Then, the synthesized image datais subjected to expansion and complement processing so that the cutsynthesized image is restored to the original image size.

In Step S4019, the synthesized image data obtained in Step S4018 isoutputted to the liquid crystal display unit provided on the backsurface, etc., of the camera and displayed as a taken image on theliquid crystal display unit.

In Step S4020, the synthesized image data obtained in Step S4018 and thefirst image data are recorded in the recording circuit 117 and arecording medium (not shown).

In Step S4021, the process returns to start. When the release button isdepressed halfway and the sw1 is on in Step S4021, the process advancesin the flow sequentially again, to Steps S4001, S4002, S4003, and S4004.When the release button is fully depressed and the sw2 is on in StepS4021, the process does not return to start but waits in Step S4021.

As described above, Embodiment 4 of the present invention has thefollowing effects in addition to the effects described above inEmbodiments 1 through 3.

In the camera of Embodiment 4, the first image (with illumination light)and the second image (without illumination light) are compared, and whenthe difference of the brightness between these is smaller than apredetermined value, the operation of the brightness adjusting circuit21 is prohibited by the brightness adjustment prohibiting circuit 31.Thereby, when illumination light does not reach the main object such asa person, the second image data are also used in addition to the firstimage data, whereby the brightness (exposure) of the main object iscomplemented.

On the other hand, when the illumination light is irradiatedsufficiently in the entire region of the first image, the imagesynthesis prohibiting circuit 32 prohibits the synthesis operation ofthe image synthesis circuit 118, so that the synthesized image can berepressed from becoming overexposed.

Furthermore, in a camera which apparently corrects exposure of a takenimage by synthesizing the first image (with illumination light) and thesecond images (without illumination light), by deleting the second imagedata after completion of the image synthesis, the capacity of therecording memory is saved and many taken image data can be recorded.

The embodiments mentioned above are examples of the present invention,and the invention is carried out by variously altering and improving theembodiments.

The present invention is applicable to a lens-integrated camera or acamera to which a lens unit is attached. A camera provided with anillumination unit 10 is described in each embodiment mentioned above,however, the present invention is also applicable to a camera to whichan illumination apparatus is attached. Namely, by making theillumination apparatus to emit the illumination light through thecommunications between the illumination apparatus and the camera, theoperations of the embodiments described above can be carried out.

Furthermore, the operations of the embodiments described above can becarried out by a program, which can be stored in a storing medium, etc.

In the Embodiments 1 to 4, the synthesized image data is generated inthe camera.

Here, it is also possible that the first and second image data are sentto an image processing apparatus such as a PC (Personal Computer) fromthe camera and the synthesized image data is generated in the imageprocessing apparatus. In this case, an operation for generating thesynthesized image in the image processing apparatus is the same as thatof the Embodiments 1 to 4.

While preferred embodiments have been described, it is to be understoodthat modification and variation of the present invention may be madewithout departing from scope of the following claims.

1. An image taking apparatus comprising: (a) an image pickup devicewhich captures at least three images sequentially; (b) a detection unitwhich detects shift amounts between a first image among the at leastthree images and each of the other images; (c) a coordinate conversionunit which performs coordinate conversion so that the other imagesoverlap the first image based on a detection result in the detectingunit; (d) a synthesis unit which synthesizes the first image and theother images as one synthesis image by performing the coordinateconversion in the coordinate conversion unit; and (e) an adjusting unitwhich darkens brightness in a first region of each of the other images,the first region corresponding to a region irradiated with light in thefirst image, wherein the first image among the at least three images iscaptured by using an illumination unit, and the other images arecaptured without using the illumination unit and are at least twocaptured images.
 2. The image taking apparatus according to claim 1,wherein the detection unit compares between the first image and theother images and detects the shift amount based on a specific point in aregion where brightness change is lower than a predetermined value. 3.The image taking apparatus according to claim 1, wherein the adjustingunit darkens the brightness in the first region of each of the otherimages by zeroing the brightness or masking the first region.
 4. Theimage taking apparatus according to claim 3, wherein the adjusting unitadjusts brightness so as to shade-off a boundary between the firstregion and a second region which is a region other than the first regionin each of the other images.
 5. The image taking apparatus according toclaim 4, wherein the adjusting unit changes brightness of the boundaryin stages or continuously.
 6. The image taking apparatus according toclaim 1, further comprising a synthesis prohibiting unit which prohibitsan image synthesis operation of the synthesis unit when brightnesschange at a specific region in each of the first image and the otherimages.
 7. The image taking apparatus according to claim 1, wherein theirradiation light is used when a last image among a plurality of imagesis taken.
 8. The image taking apparatus according to claim 1, furthercomprising a recording unit, in which stores the first image and thesynthesized image are recorded.
 9. A control method of an image takingapparatus comprising an image pickup device capturing an image,comprising the steps of: (a) capturing at least three imagessequentially by the image pickup device; (b) detecting shift amountsbetween a first image among the at least three images and each of theother images; (c) performing coordinate conversion so that the otherimages overlap the first image based on a detection result in thedetecting unit; (d) synthesizing the first image and the other images asone synthesis image by the coordinate conversion in the performing step;and (e) darkening brightness in a first region of each of the otherimages, the first region corresponding to a region irradiated with lightin the first image, wherein the first image among the at least threeimages is captured by using an illumination unit and the other imagesare captured without using the illumination unit, the other image beingat least two captured images.
 10. An image processing apparatus whichprocesses at least three images which are captured by an image pickupdevice sequentially, a first image among the at least three images beingcaptured by using an illumination unit, the other images being capturedwithout using the illumination unit and being at least two images, theimage processing apparatus comprising: (a) a detection unit whichdetects shift amounts between the first image and each of the otherimages; (b) a coordinate conversion unit which performs coordinateconversion so that the other images overlap the first image based on adetection result in the detecting unit; (c) a synthesis unit whichsynthesizes the first image and the other images as one synthesis imageby performing the coordinate conversion in the coordinate conversionunit; and (d) an adjusting unit which darkens brightness in a firstregion of each of the other images, the first region corresponding to aregion irradiated with light in the first image.
 11. An image processingmethod for processing at least three images which are captured by animage pickup device sequentially, a first image among the at least threeimages being captured by using an illumination unit, the other imagesbeing captured without using the illumination unit and being at leasttwo images, the image processing method comprising the steps of: (a)detecting shift amounts between the first image and each of the otherimages; (b) performing coordinate conversion so that the other imagesoverlap the first image based on a detection result in the detectingunit; (c) synthesizing the first image and the other images as onesynthesis image by the coordinate conversion in the performing step; and(d) darkening brightness in a first region of each of the other images,the first region corresponding to a region irradiated with light in thefirst image.
 12. A medium in which a program for processing at leastthree images which are captured by an image pickup device sequentiallyis stored, a first image among the at least three images being capturedby using an illumination unit, the other images being captured withoutusing the illumination unit and being at least two images, the programcomprising the steps of: (a) detecting shift amounts between the firstimage and each of the other images; (b) performing coordinate conversionso that the other images overlap the first image based on a detectionresult in the detecting unit; (c) synthesizing the first image and theother images as one synthesis image by the coordinate conversion in theperforming step; and (d) darkening brightness in a first region of eachof the other images, the first region corresponding to a regionirradiated with light in the first image.